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1.0  Exclusive  Summary 


This  report  describes  in  detail  the  procedures  and  approaches  WRT  Inc.  team  took  to  complete  an 
Environmental  Security  Technology  Certification  Program  (ESTCP)-funded  live-site  UXO 
demonstration  study  at  the  former  Waikoloa  Maneuver  Area  (WMA)  in  Waikoloa,  Hawaii,  under 
ESTCP  Munitions  Response  Project  201227.  The  main  objectives  of  this  demonstration  study  at 
the  former  WMA  were: 

1.  Test  the  discrimination  capability  of  the  advanced  EMI  models  for  live-site  conditions  in  the 
presence  of  magnetically  susceptible  soils. 

2.  Validate  technology  by  inverting  the  target’s  intrinsic  parameters  and  identifying  robust 
classification  features,  which  distinguish  UXO  targets  from  non-hazardous  objects. 

3.  Document  the  applicability  and  limitations  of  the  advanced  EMI  discrimination  technologies 
for  magnetically  susceptible  soils. 

Parsons  delivered  all  WMA  cued  (1032)  and  site  specific  background  data  set  to  us.  All  data  were 
processed  using  our  advanced,  electromagnetic  induction  (EMI)  models,  such  as  orthonormalized 
volume  magnetic  source  (ONVMS),  joint  diagonalization  (JD)  and  differential  evolution  (DE) 
approach.  Our  studies  showed  that  the  ONVMS  method  is  able  to  separate  targets  responses  from 
magnetic  soil  responses  without  any  difficulties  when  the  distance  between  target  and  sensor  is 
less  than  30  cm.  The  prioritized  dig  list  was  generated  and  submitted  to  the  ESTCP  office  and 
scored  against  the  ground  truth  from  the  intrusive  investigation.  Two  seed  items  were  incorrectly 
classified  as  nonhazardous  clutter.  Failure  analysis  indicated  that  the  incorrect  classification  was 
due  to  the  combination  of  lateral  offset,  strong  geological  background  responses  and  breakdown 
of  one  of  the  receiver  cubes.  Namely,  misclassification  of  one  of  the  two  targets  was  due  to 
significant  (43  cm)  offset  between  the  MM  data  collection  point  and  the  actual  location  of  the  seed. 
The  second  incorrect  classification  appeared  to  be  a  result  of  combination  of:  a)  the  large  offset 
(27.86  cm)  between  the  MM  center  and  the  seed  target,  and  b)  being  the  completely  failed  Rx  cube 
(Rx#0)  adjacent  to  the  actual  seed  target.  We  believe,  that  having  a  robust  in-field  or  off-line 
quality  check  step  could  guide  an  operator  to  place  the  sensor  close  to  the  anomaly,  collect  high 
quality  data  and  avoid  these  misclassifications  in  challenging  sites,  such  as  WMA. 

Comparisons  between  our  classification  and  Parsons  intrusive  investigation  results  revealed 
significant  discrepancies.  For  example,  Parsons  considered  anomalies  #29,  36,  199,  441,  and  442 
as  “no  contacts”,  however  MM  data  and  our  classification  results  show  with  high  confidence  that 
these  anomalies  were  compact,  37  mm  projectiles  and  60  mm  mortar  like  metallic  targets.  These 
discrepancies  indicate  that  the  intrusive  procedure,  which  was  used  by  Parsons  at  WMA,  failed  to 
document  accurately  all  intrusive  results,  or  failed  to  detect  and  clear  all  UXO  like  targets  on  the 
site.  Thus,  for  cleaning  UXO  sites  reliably  and  effectively  the  classification  results  must  be  used 
to  guide  and  validate  intrusive  investigation  results. 
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2.0  INTRODUCTION 


This  demonstration  is  designed  to  illustrate  discrimination  performance  of  advanced, 
electromagnetic  induction  (EMI)  models,  such  as  orthonormalized  volume  magnetic  source 
(ONVMS)  [1],  joint  diagonalization  (JD)  and  differential  evolution  (DE)  approach  [2],  [3]-[5],  at 
the  former  Waikoloa  Maneuver  Area  (WMA),  Hawaii,  as  a  part  of  ESTCP  live-site  UXO 
classification  pilot  studies  program.  The  site  -  located  on  the  northwest  side  of  the  Island  of 
Hawaii,  approximately  30  miles  north  of  the  city  of  Kailua-Kona  in  the  South  Kohala  District  - 
was  acquired  by  the  Navy  in  1943  and  used  as  a  military  training  camp  and  artillery  range  for 
50,000  troops  until  1945.  The  advanced  EMI  models  used  here  have  been  developed  under  the 
SERDP  MM-1572  project  and  successfully  applied  to  the  next  generation  EMI  sensor’s  [6]-[12] 
datasets  collected  at  the  Camp  Sibert,  AL,  the  Aberdeen  Proving  Ground  (APG),  San  Luis  Obispo, 
CA[7],  Camp  Butner,  NC  [9],  [13]  and  Camp  Beale,  CA  [10]  live  sites.  In  this  demonstration  the 
models  are  applied  to  data  collected  using  the  Metal  Mapper  (MM)  instrument.  The  suspected 
munitions  at  the  site  are  60-mm  and  80-mm  high  explosive  mortars,  75-mm,  105-mm,  and  155- 
mm  projectiles,  2.36-inch  rocket  propelled  anti-tank  rounds,  US  MK  II  hand  grenades,  Rockets, 
Ml  anti-tank  land  mines,  and  Japanese  ordnance.  During  WMA  MM  data  analysis  and 
classification  study,  for  each  anomaly,  MM  data  are  inverted  and  the  targets  intrinsic  (total  volume 
magnetic  source  (NVMS)  i.e.  the  size,  shape  and  material  properties)  and  extrinsic  (location, 
depth,  orientation)  parameters  are  estimated.  The  intrinsic  parameters  are  used  for  classification 
and  a  ranked  dig-list  is  generated.  The  dig-list  is  submitted  to  the  Institute  for  Defense  Analyses 
(IDA)  for  independent  scoring.  The  advanced  models  classification  performances  are  assessed  and 
documented  based  on  the  independently  scored  results. 

2.1  Objective  of  the  Demonstration 

The  principal  objectives  of  this  project  are  to  apply  advanced  EMI  models  to  UXO  discrimination 
problems  on  live  sites  and  to  demonstrate  classification  capability  under  real  world  scenarios. 
Specific  technical  objectives  are  to: 

4.  Demonstrate  the  discrimination  capability  of  the  advanced  EMI  models  for  live-site 

conditions; 

5.  Invert  the  target’s  intrinsic  parameters,  and  identify  robust  classification  features,  which 

distinguish  UXO  targets  from  non-hazardous  objects.  Namely,  the  technology  should: 

a.  Identify  all  seeded  and  native  UXO. 

b.  Eliminate  at  least  75%  of  targets  that  do  not  correspond  to  targets  of  interests  (TOI). 

6.  Identify  sources  of  uncertainty  in  the  classification  process  and  include  them  in  a  dig/no-dig 
decision  process; 

7.  Understand  and  document  the  applicability  and  limitations  of  the  advanced  EMI 
discrimination  technologies  in  the  context  of  project  objectives,  site  characteristics,  and 
suspected  ordnance  contamination. 
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3.0  TECHNOLOGY 


The  advanced  EMI  models  with  a  statistical  signal  processing  approach,  developed  and  tested  over 
the  past  six  years  as  part  of  SERDP  MM- 1572,  showed  excellent  discrimination  performance  when 
applied  to  next-generation-sensor  data  collected  at  various  live  sites,  such  as  Camp  Sibert,  Al,  San 
Louis  Obispo  (SLO),  CA,  Camp  Butner,  NC  and  Camp  Beale,  as  well  as  APG  test  sites.  The 
technology  was  able  to  single  out  UXO  ranging  in  caliber  from  25  mm  up  to  155  mm.  The 
ONVMS/JD  technique,  moreover,  was  seen  to  provide  excellent  classification  in  both  single  and 
multiple-target  scenarios  when  combined  with  multi-axis/transmitter/receiver  sensors  like 
TEMTADS  [1]  and  the  MM  [6],  The  methodology,  augmented  to  include  a  suite  of  classifiers, 
was  also  adapted  to  handheld  sensors  like  the  MPY  and  2x2-3D  TEMTADS  [12]  .Recent 
classification  studies  showed  that  not  only  were  the  advanced  EMI  models  able  to  classify  all  “easy 
seed  UXO  items,”  they  also  managed  to  identify  all  other  targets,  no  matter  how  unexpected  or 
site-specific,  and  as  small  as  3-cm  fuzes  [11].  Detailed  Description  of  data  inversions,  targets 
feature  clustering  and  selections  and  classicization  using  the  ONVMS,  JD,  DE  and  Gaussian 
mixture  model  approaches  are  documented  in  reports  [11],  [28],  and  peer-reviewed  publications 
[5],  [14],  [15],  [29], 


3.1.1  WMA  MM  data  inversion  and  classification  scheme 

The  MM  sensor’s  Tx  and  Rx  signals  detailed  modeling  approach  using  the  ONVMS-DE  algorithm 

is  described  in  [11], 

Step  1.  Data  pre-processing:  All  MM  data  were  pre-processed  using  a  Matlab  Code  (see 
Appendixes  in  [11]).  The  code  reads  comma-delimited  format  CSV  files,  subtracts 
backgrounds  from  data  and  transfers  them  to  ASCII  files  compatible  with  the  ONVMS- 
DE  code  (ONVMS_MM.exe).  The  user  needs  only  to  specify  the  path  to  the  folder  with 
the  CSV  files;  the  code  then  converts  them  all  and  prepares  for  inversion. 

Step  2.  Create  MM  Multi  Static  Response  (MSR)  data  matrix:  Using  procedures  described  in  [11], 
we  construct  the  measurement  matrix  H(tq) for  each  anomaly  and  use  it  to  create  the  MM 
MSR  data  matrix. 

Step  3.  Eigenvalue  analysis:  The  JD  technique  is  applied  to  the  created  MM  MSR  data  matrix  to 
extract  the  time-dependent  eigenvalues  for  each  anomaly.  The  eigenvalues  for  some  of 
the  WMA  seeded  anomalies  and  soils  are  depicted  in  Figure  1  and  Figure  2.  The  MSR 
data  matrix  eigenvalues  are  intrinsic  properties  of  the  targets;  each  target  has  at  least  three 
eigenvalues  above  the  threshold  (noise  level:  low  magnitude  eigenvalues).  For  example, 
Figure  1  shows  the  eigenvalues  extracted  for  a  37mm  projectile,  a  small  ISO,  a  60  mm 
mortar  and  a  81  mm  projectile,  and  Figure  2  shows  eigenvalues  vs  time  for  soils.  The 
results  illustrate  that  each  target  has  distinguishable  eigenvalues  that  can  be  used  for 
classification;  Note  that  the  magnitudes  of  the  MSR  eigenvalues  depend  on  the  depths  and 
orientations  of  the  targets  [2];  therefore,  the  user  should  use  only  their  shapes  when 
performing  classification.  In  this  study,  we  examined  the  eigenvalues  versus  time  for  each 
case  and  used  them  to  estimate  the  number  of  sources  and  SNR.  Once  we  had  the  JD 
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analysis  and  estimated  SNR  for  each  anomaly  we  proceeded  to  invert  all  cued  MM 
datasets  using  the  combined  ONVMS-DE  algorithm  for  multi-targets. 


106 1— - - - - — . . . —  ■  106 


Time  [nsec]  Time  ^sec1 


- Case-00721 

- Case-00107 

Figure  1.  WMA  MM  multi-static  response  matrix  eigenvalues  versus  time  for  some  samples 
of  seeded  anomalies. 


MR-201227  Advanced  EMI  Models  for  Live-site 
UXO  Discrimination  -  Waikoloa,  HI 


4 


December,  2015 


Eigen  Values  [Arb] 


103 

Time  [psec] 


10 


Time  [psec] 


Figure  2.  MM  multi-static  response  matrix  eigenvalues  versus  time  for  some  samples  of 
WMA  soils. 


Time 

Figure  3.  Inverted  total  ONVMS  time-decay  profiles  for  a  WMA  medium  ISO  target. 


MR-201227  Advanced  EMI  Models  for  Live-site 
UXO  Discrimination  -  Waikoloa,  HI 


5 


December,  2015 


Step  4.  Extract  the  total  ONVMS  for  each  anomaly.  The  targets’  extrinsic  and  intrinsic  parameters, 
including  the  total  ONVMS  where  extorted  for  all  anomalies. 
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Figure  4.  Scatter  plot  of  size  and  decay  for  all  WMA  anomalies  based  on  the  extracted  total 
ONVMS  for  time  channels  N#  5,  15,  25,  and  35. 


Step  5.  Create  a  custom  training  list.  We  use  the  size  i.e.  magnitude  of  the  inverted  primary 
effective  polarizability  Mzz  (ti)  (Mzz  (t)=(ONVMSzz(t))  at  the  first  time  channel  and  decay  i.e. 
ratio  of  the  primary  effective  polarizability  at  the  n-th  time  channel  Mzz  (tn)  to  that  at  the  Mzz 
(ti)  as  classification  feature  parameters  and  determined  the  best  separations.  The  values  of 
logl0 [M„ (tn )  /  M__  (t, )]  versus  log|f| are  plotted  for  all  WMA  MM  data  sets,  at  the  5th, 

15th,  25th,  and  35th  time  channels.  Visual  examination  shows  that  there  are  no  distinguishable 
clusters  at  the  5th  channel;  at  later  times,  on  the  other  hand,  the  decay-vs.-size  distribution 
starts  to  cluster.  We  used  the  features  evaluated  at  the  35  th  time  channel  and  applied  statistical 
classification  techniques  [11],  namely  data  were  clustered  using  Euclidean  distance  and  Ward 
linkage  (for  more  info  see  [13]).  For  each  cluster  we  computed  the  centroid  and  determined 
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the  anomaly  closest  to  it.  We  included  this  anomaly  in  the  custom  training  data  list.  The 
clustering  results  of  all  WMA  MM  anomalies  are  depicted  in  Figure  5.  Here  each  circle  is 
centroid  of  a  cluster  and  dots  with  the  same  color  in  and  around  the  circle  belongs  to  a  cluster; 
In  addition  to  the  statistical  clustering  algorithm,  ONVMS  time  decay  curves  were  inspected 
for  each  anomaly:  we  used  the  TONVMS  time  decay  shapes  and  symmetries  to  further  validate 
or  modify  the  custom  training  anomaly  list.  Anomalies  with  significantly  asymmetric 
TONVMS  were  removed  from  the  training  list;  anomalies  with  fast  decay  but  symmetric 
profiles  were  added  to  the  training  list  for  which  we  requested  the  identifying  ground  truth. 
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Figure  5.  Result  of  the  clustering  for  the  WMA  anomalies  using  the  size  and  shape  information  for 
n  =  35.  The  circles  denote  the  anomalies  for  training. 

Step  6.  Request  ground  truth  for  selected  anomalies.  The  custom  training  list,  a  combination  of 
JD,  clustering  and  ONVMS-DE  single-target  inversion  results,  was  submitted  to  the 
ESTCP  office,  which  then  provided  the  ground  truth  for  training.  We  used  the  delivered 
ground  truth  to  identify  the  different  possible  TOI  types  and  their  size  variations.  There 
were  seeded  small  and  medium  size  ISO-s,  37  mm  and  81  mm  projectiles  and  60  mm 
mortars  which  the  ESTCP  office  identified  as  TOI. 
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Figure  6.  Inverted  total  ONVMS  time-decay  profiles  for  a  small  (top  figure)  and  medium 
(bottom  figure)  WMA  ISO  targets. 
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Figure  7.  Inverted  total  ON  VMS  time-decay  profiles  for  WMA  37  mm  projectiles  with  (top 
figure)  and  without  (bottom  figure)  copper  band. 
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Figure  8.  Inverted  total  ON  VMS  time-decay  profiles  for  WMA  60  mm  mortar  without  (top 
figure)  and  with  (bottom  figure)  tail. 
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Figure  9.  Inverted  total  ONVMS  time-decay  profiles  for  WMA  81  mm  projectiles. 


Step  7.  Create  ranked  dig  list.  Armed  with  the  ground  truth  of  (a  total  of  87)  custom  identified 
training  anomalies  and  the  inverted  total  ONVMS  for  each  MM  cued  data,  we  created  a 
library  for  all  TOI.  The  inverted  total  ONVMS  for  the  anomalies  that  were  classified  as 
TOI  appear  in  Figure  6-Figure  10.  All  the  inverted  total  ONVMS  are  seen  to  cluster 
reasonable  well,  and  each  target  has  a  total  ONVMS  with  features  that  make  it  easy  to 
identification:  its  amplitude  at  the  first  time  channel,  its  decay  rate,  or  the  separation 
between  the  primary  (green  lines)  and  secondary  (black  and  blue)  components  at  different 
time  channels.  It  is  worth  noting  that  the  inverted  total  ONVMS  for  37  mm  projectiles  are 
clustered  more  closely  than  for  other  TOI  targets.  To  classify  targets  accurately,  we  used 
a  classification  approach  based  on  symmetry  and  time-decay  shapes,  and  ranked  some  of 
symmetric  targets  as  TOIS. 
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Figure  10.  Inverted  total  ONVMS  time-decay  profiles  for  2.36”  rocket  part,  which  was 
identified  as  TOI. 

Step  8.  Submit  the  dig  list  to  ESTCP.  The  final  prioritized  dig  list  was  submitted  to  the  Institute 
for  Defense  Analyses  (IDA)  for  independent  scoring.  The  scored  results  were  sent  back  in 
the  form  of  a  receiver  operating  characteristic  (ROC)  curve,  which  we  depict  in  Figure  11. 
The  result  shows  that  of  the  87  targets  that  were  dug  for  training,  57  targets  were  not  TOI 
(shift  along  x-axis)  and  30  were  (shift  along  y-axis);  all  TOI  targets  were  ranked  as  dig 
except  one  small  ISO  and  one  60  mm  mortar,  which  laterally  were  offset  more  or  about 
than  30  cm  from  the  center  of  MM  sensor.  The  result  clearly  shows  that  advanced  EMI 
sensing  and  classification  technologies  can  be  applied  to  active  and  challenge  UXO  sites, 
which  in  addition  to  man-made  clutter  also  consist  magnetic  soils.  Our  studies  show  that 
even  when  targets  are  buried  in  a  highly  susceptible  magnetic  soil,  the  technology  provides 
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the  ability  to  leave  at  least  80%  of  clutter  items  in  the  ground  if  the  lateral  offset  between 
the  sensor’s  center  and  acquired  target  is  less  than  30  cm. 


MM 


Number  of  Non-TOIs  Incorrectly  Classified 


Figure  11.  ROC  curve  for  the  WMA,  HI  MM  data. 
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3.2  A  Brief  Chronological  Summary 


The  basic  concepts  of  the  Advanced  EMI  models  have  evolved  largely  from  methodologies 
developed  over  the  last  eleven  years  by  the  Electromagnetic  Sensing  Group  at  Dartmouth  College 
in  close  collaboration  with  the  ERDC-CRREL  team.  The  developments  were  supported  by  various 
SERDP  projects.  In  2007,  SERDP  awarded  Project  MM-1572,  “A  Complex  Approach  to  UXO 
Discrimination:  Combining  Advanced  EMI  Forward  and  Statistical  Signal  Processing”  to  Sky 
Research,  Inc.  This  project  supported  the  development  and  implementation  of  NSMS,  an  advanced 
and  physically  complete  EMI  model,  and  its  combination  and  integration  with  statistical 
classification  algorithms  such  as  neural  networks,  support  vector  machines,  and  Gaussian  mixture 
model  clustering.  These  methods  were  tested  at  the  APG,  Camp  Sibert,  and  SLO  live  sites.  The 
NSMS  method  was  extended  into  the  orthonormalized  volume  magnetic  source  (ONYMS) 
technique.  The  basics  of  the  ONVMS/Joint  Diagonalization  technique  were  developed  under  the 
following  SERDP  projects:  “Electromagnetic  Induction  Modeling  for  UXO  Detection  and 
Discrimination  Underwater/Multi  Target  Inversion  and  Discrimination”  (MM-1632,  Dartmouth 
College)  and  “Isolating  and  Discriminating  Overlapping  Signatures  in  Cluttered  Environments” 
(MM-1664,  a  joint  project  between  Dartmouth  College  and  USACE-CRREL).  The  ONVMS  and 
JD  approaches  were  tested  at  Camp  Butner  and  Camp  Beale  under  Projects  SERDP  MR- 1572  and 
ESTCP  MR-201 101,  respectively. 
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4.0  PERFORMANCE  OBJECTIVES 


The  performance  objectives  of  this  ESTCP  live-site  discrimination  study  are  to  achieve  high 
probability  of  discrimination  of  UXO  from  a  wide  range  of  clutter  items,  to  process  all  data  sets, 
to  minimize  number  of  Can’t  analyze/Can’t  decide  targets,  to  minimize  number  of  false  positives, 
and  to  identify  all  UXO  with  high  confidence.  The  performance  objectives  are  summarized  in 

Table  1. 


Table  1.  Performance  Objectives 


Performance 

Objective 

Metric 

Data  Required 

Success  Criteria 

Maximize  correct 
classification  of 
munitions 

Number  of  targets-of- 
interest  retained 

•  Prioritized  anomaly 
lists 

•  Scoring  reports  from 
Institute  for  Defense 
Analyses  (IDA) 

Approach  correctly 
classifies  all  targets-of- 
interest 

Maximize  correct 
classification  of  non¬ 
munitions 

Number  of  false  alarms 
eliminated 

•  Prioritized  anomaly 
lists 

•  Scoring  reports  from 
IDA 

Reduction  of  false 
alarms  by  >75%  while 
retaining  all  targets  of 
interest 

Specification  of  no-dig 
threshold 

Probability  of  correct 
classification  and 
number  of  false  alarms 
at  demonstrator 
operating  point 

•  Demonstrator- 
specified  threshold 

•  Scoring  reports  from 
IDA 

Threshold  specified  by 
the  demonstrator  to 
achieve  the  above 
criteria 

Minimize  number  of 
anomalies  that  cannot 
be  analyzed 

Number  of  anomalies 
that  must  be  classified 
as  “Unable  to  Analyze” 

•  Demonstrator  target 
parameters 

Reliable  target 
parameters  can  be 
estimated  for  >  90%  of 
anomalies  on  each 
sensor’s  detection  list 

Correct  estimation  of 
target  parameters 

Accuracy  of  estimated 
target  parameters 

•  Demonstrator  target 
parameters 

•  Results  of  intrusive 
investigation 

Total  ONVMS  ±  10% 

X,  Y  <  10  cm 

Z  <  5  cm 
size  ±  10% 
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4.1  Objective:  maximize  correct  classification  of  munitions 

The  effectiveness  goal  of  the  technology  for  discrimination  of  munitions  is  maximizing  correct 
classification  of  targets  of  interest  vs.  non-TOI  with  high  (95%)  confidence.  The  confidence 
interval  is  calculated  from  a  partial  (up  to  dig  stop  point)  ROC. 

4.1.1  Metric 

Identify  all  seeded  and  native  TOI  with  high  confidence  using  advanced  EMI  discrimination 
technologies.  Our  estimates  were  based  on  using  the  extracted  total  ONVMS  as  input  to  statistical 
and  template  matching  algorithms. 

4.1.2  Data  requirements 

We  analyzed  MM  data  and  identified  custom  training  data  sets  (using  less  than  <5  %  of  entire 
data).  We  requested  the  ground  truth  for  the  custom  training  data  sets  and  used  them  to  validate 
the  models.  We  generated  a  dig  list  that  was  scored  by  IDA. 

4.1.3  Success  criteria  evaluation  and  results 

The  objective  was  considered  to  be  met  if  all  seeded  and  native  UXO  items  could  be  identified 
below  an  analyst-specified  no-dig  threshold. 

4.1.4  Results 

Our  algorithm  was  able  to  classify  all  TOI-s  correctly,  but  two  TOIs:  one  small  ISO  and  one  60 
mm  mortar.  Figure  1 1  shows  the  ROC  curves  obtained  for  all  WMA  anomalies. 

4.1.5  Root  cause  analysis 

To  understand  the  cause  of  miss-classifying  these  two  TOIs  we  re-examined  classification 
procedures  for  all  WMA  anomalies.  Classification  studies  were  conducted  at  three  TO  17  and  T20- 
A  and  T020-B  areas.  These  TO  17  and  T020  areas  were  chosen  to  understand  how  lava  flows  under 
them  effect  on  geological  background  responses  and  targets  classifications.  The  T017  site  is  near 
the  coast  and  the  lava  flow  underlying  it  is  considerable  younger,  than  the  lava  flow  underlying 
the  two  T020  sites.  The  both  missed  TOIS  were  in  area  T017.  To  assess  geological  background 
responses,  we  analyzed  largest  eigenvalues  versus  time  for  soils  in  these  areas  when  the  sensor  is 
placed  on  the  ground.  The  results  in  Figure  12  show  that  geological  background  response  for  the 
TO  17  area  is  higher  than  background  responses  for  area  T020A  and  T020B.  Usually  high 
background  responses  degrade  data  and  causes  miss-classifications,  particularly  when  targets 
lateral  offsets  are  more  or  about  30  cm  from  the  center  of  the  sensor. 
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Figure  12.  Eigen  values  vs  time  for  three  TO  17,  T020  A  and  T020B  areas. 


In  addition  to  geological  background  noises,  a  sensor’s  malfunction  significantly  effects  on  targets 
classification  as  well,  particularly  when  the  failed  sensor  is  close  to  the  targets.  Unfortunately, 
during  the  WMA  MM  data  collection  one  of  the  receivers  failed  completely.  Namely,  all  three 
components  of  Rx  #0  did  not  work  properly,  and  fields  measured  by  the  Rx  #0  were  not  included 
in  data  inversion  and  processing.  The  inverted  total  ONVMS  (effective  polarizabilities)  and 
eigenvalues  for  the  miss-classified  small  ISO  (anomaly  #1027)  and  60  mm  mortar  (anomaly# 
1047)  are  depicted  on  Figure  13.  The  inverted  effective  polarizabilities  for  the  anomaly  #1027 
shows  some  symmetry  but  its  magnitude  is  much  smaller  than  for  a  library  ISO  target,  see  Figure 
6.  Also,  eigenvalues  versus  time  shows  that  there  is  a  significant  soil  response  (eigenvalues  linear 
decay  in  log-log  plot,  see  Figure  12)  as  well.  To  better  understand  why  the  inverted  effective 
polarizabilities  for  anomaly  #1027  do  not  match  the  effective  polarizabilities  for  a  library  small 
ISO  target,  we  checked  the  distance  between  the  MM  sensor’s  center  and  dug  target  location.  The 
analysis  show  that  for  the  anomaly  #1027,  the  sensor’s  center  was  located  28  cm  from  the  target. 
Since  this  offset  is  less  than  30  cm,  the  extracted  classification  parameters  should  have  been  robust 
and  closely  correlated  with  the  effective  polarizabilities  for  a  library  small  ISO-s  target.  However, 
after  further  investigation  it  was  found  that  the  closest  Rx  sensor  to  the  target  was  the  failed  Rx  #0 
sensor.  This  and  the  dominant  ground  responses  in  other  sensors  cause  the  algorithm  to  miss- 
classify  this  small  target.  We  believe,  that  positioning  the  center  of  the  sensor  closer  to  the 
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anomaly,  or  having  properly  working  Rx  adjacent  to  the  target  could  have  avoided  this 
misclassification. 

For  seed  WK-1047,  the  sensor  was  located  43  cm  from  the  target.  This  offset  is  significantly 
outside  the  30  cm  objective  radius  and,  as  a  result  the  extracted  polarizabilities  values  are  high  but 
not  symmetric,  decay  linearly  in  log-log  scale  and  looks  like  more  a  soil’s  characteristic,  then 
signals  from  a  compact  60  mm  mortar  target.  While  in  case  of  non-permeable  soil  conditions,  our 
algorithms  were  able  to  obtain  accurate  classification  features  at  more  than  30  cm  offsets,  we 
believe  the  presence  of  a  significant  magnetic  ground  response  made  it  very  difficult  to  extract 
robust  classification  features  with  the  sensor  at  this  43  cm  location  for  anomaly  #1047. 


Time  [psec]  Time  [psec] 


Figure  13.  Total  ONVMS  and  eigenvalues  for  missed  small  ISO  (anomaly  #1027)  and  60  mm  mortar 
(anomaly  #1047)  targets; 
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Figure  14.  Photos  of  intrusive  investigated  anomalies  #29,  #36,  #199,  #441,  #442. 
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4.1.6  Mystery  targets:  discrepancies  between  classification  and  intrusive  results 

All  intrusive  operations  at  WMA  site  were  conducted  by  Parsons’s  Personnel.  According  to  the 
report  MR-201 1 04-DR- Waikoloa  [30]:  Parsons  used  the  Minelab  Explorer  SE  to  determine  the 
initial  approach  to  every  target  and  as  a  screening  process  to  assess  if  metal  was  present  in  the 
subsurface  or  if  the  anomaly  was  caused  by  the  local  geology.  If  the  Minelab  Explorer  SE 
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Figure  15.  Top  row:  comparisons  between  total  ONVMS  for  a  library  37  mm  projectile 
(red  lines),  with  copper  band,  and  for  anomalies  #36  and  #442.  Bottom  row  eigenvalues 
versus  time  for  anomalies  #36  and  #442. 
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indicated  that  there  was  no  metal  present  in  the  area,  the  anomaly  was  considered  a  “no  contact.” 
A  GPS  point  was  then  taken  at  the  location  of  the  flag  and  a  photograph  was  collected  of  the  area 
surrounding  the  flag.  If  the  Minelab  Explorer  SE  indicated  that  there  was  metal  present  in  the 
subsurface,  then  the  UXO  technicians  excavated  the  item.  Location  data  captured  by  GPS  were 
used  to  document  the  center  mass  and  depth  of  each  item.  A  photograph  was  collected  of  the  item; 
Lastly,  an  EM61  unit  was  used  to  scan  the  location  to  confirm  the  absence  of  all  metallic  items 
from  that  target  location  or  that  the  pre-millivolt  reading  had  been  reduced  by  at  least  75%. 


Time  [jisec] 
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Figure  16.  Top  row:  comparisons  between  total  ONVMS  for  a  library  37  mm  projectile  (red  lines),  without 
copper  band,  and  for  anomalies  #29  and  #441.  Bottom  row  eigenvalues  versus  time  for  anomalies  #29  and 


#441. 
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Figure  17.  Top  row:  comparisons  between  total  ONVMS  for  a  library  60  mm  projectile  (red 
lines  and  for  anomaly  #199.  Bottom  row  eigenvalues  versus  time  for  anomaly  #199. 


Using  the  above  described  intrusive  investigation  approach,  Parsons  considered  anomalies  #29, 
36,  199,  441,  and  442  as  “no  contacts”  see  Figure  14.  However,  MM  data  and  our  classification 
results  depicted  in  Figure  15,  Figure  16  and  Figure  17  clearly  show  that  there  are  compact  metallic 
targets  (see  eigenvalues  vs  time  and  total  ONVMS).  The  effective  polarizabilities  of  these 
anomalies  closely  match  the  polarizabilities  of  library  37  mm  projectiles  and  60  mm  mortar, 
indicating  that  the  intrusive  procedure  failed  to  document  all  detected  targets  correctly,  or  failed 
detect  and  clear  all  hazardous  targets  on  the  site.  If  later  occurred,  then  we  believe  that  the 
MineLab  explorer  SE  is  not  a  suitable  system  for  metallic  targets  detection  in  magnetic  soil,  and 
for  accurate  cleaning  UXO  sites  the  classification  results  must  be  used  to  guide  and  validate 
intrusive  investigation  results. 
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4.2  Objective:  maximize  correct  classification  of  non-munitions 

The  technology  aims  to  minimize  the  number  of  false  negatives,  i.e.  maximize  the  correct 
classification  of  non-TOI. 

4.2.1  Metric 

We  compared  the  number  of  non-TOI  targets  that  can  be  left  in  ground  with  high  confidence  using 
the  advanced  EMI  discrimination  technology  to  the  total  number  of  false  targets  that  would  be 
present  if  the  technology  were  absent. 

4.2.2  Data  requirements 

This  objective  requires  the  prioritized  anomaly  list,  which  was  created  by  our  team,  and 
independent  scoring  reports  from  IDA. 

4.2.3  Success  criteria  evaluation  and  results 

The  objective  was  considered  to  have  been  met  if  the  method  eliminated  at  least  75%  of  targets 
that  did  not  correspond  to  targets  of  interest  in  the  discrimination  step. 

4.2.4  Results 

Due  to  high  background  noise,  the  sensor’s  large  lateral  offset  from  anomalies  and  failure  of  one 
of  the  sensors,  this  objective  was  not  met  successfully.  The  advanced  EMI  discrimination 
technology  was  able  to  eliminate  50%  of  non-TOI  in  the  WMA  demonstration  after  all  TOI  were 
classified  correctly. 

4.3  Objective:  specify  a  no-dig  threshold 

This  project  aims  to  provide  high  classification  confidence  approach  for  UXO-site  managers.  One 
of  the  critical  quantities  for  minimizing  UXO  residual  risk  and  providing  regulators  with 
acceptable  confidence  is  the  specification  of  a  no-dig  threshold. 

4.3.1  Metric 

We  compared  an  analyst’s  no-dig  threshold  point  to  the  point  where  100%  of  munitions  were 
correctly  identified. 

4.3.2  Data  requirements 

To  meet  this  requirement,  we  needed  scoring  reports  from  IDA. 

4.3.3  Success  criteria  evaluation  and  results 

The  objective  would  be  met  if  a  sensor-specific  dig  list  placed  all  the  TOI  before  the  no-dig  point 
and  if  additional  digs  (false  positives)  were  requested  after  all  TOI  were  identified  correctly. 
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4.3.4  Results 


This  objective  was  not  met.  All  TOI-s,  except  two,  were  located  before  the  analyst’s  no-dig 
threshold  point  see  Figure  11. 

4.4  Objective:  minimize  the  number  of  anomalies  that  cannot  be  analyzed 

Some  anomalies  may  not  be  classified,  either  because  of  the  data  are  not  sufficiently  informative — 
the  sensor  physically  cannot  provide  the  data  to  support  classification  for  a  given  target  at  a  given 
depth — or  because  the  data  processing  was  inadequate.  The  former  is  a  measure  of  instrument 
performance  for  all  anomalies  for  which  all  data  analysts  converge.  The  latter  is  a  measure  of  our 
data  analysis  quality,  where  our  target  diagnostic  differs  from  that  made  by  other  analysts. 

4.4.1  Metric 

The  metric  for  this  objective  is  the  number  of  anomalies  that  cannot  be  analyzed  by  our  method, 
and  the  intersection  of  all  anomaly  lists  among  all  analysts. 

4.4.2  Data  requirements 

Each  analyst  submitted  their  anomaly  list.  IDA  scored  all  lists  and  returned  a  list  of  anomalies  that 
could  not  be  analyzed  by  any  analyst  (“cannot  analyze”  or  “failed  classification”). 

4.4.3  Success  criteria  evaluation  and  results 

The  objective  was  met  if  at  least  95%  of  the  selected  anomalies  that  verify  the  aforementioned 
depth  requirement  could  be  analyzed. 

4.4.4  Results 

This  objective  was  successfully  met.  All  data  sets  for  all  anomalies,  except  one,  were  analyzed. 
Only  one  anomaly,  which  did  not  have  appropriate  data,  was  ranked  as  “cannot  analyze.” 

4.5  Objective:  correct  estimation  of  target  parameters 

The  combined  ONVMS-DE  algorithm  provides  intrinsic  and  extrinsic  parameters  for  the  different 
targets.  The  intrinsic  parameters  were  used  for  classification,  while  the  extrinsic  parameters  (i.e., 
target  locations)  were  utilized  for  residual  risk  assessment. 

4.5.1  Metric 

The  classification  results  entirely  depend  on  how  accurately  these  parameters  are  estimated. 

4.5.2  Data  requirements 

To  achieve  this  objective,  we  inverted  and  tabulated  the  intrinsic  and  extrinsic  parameters  for  all 
targets.  To  validate  the  extracted  extrinsic  parameters,  we  needed  the  results  of  intrusive 
investigations. 
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4.5.3  Success  criteria  evaluation  and  results 

The  objective  was  met  if  the  targets’  intrinsic  parameters  varied  within  ±10%,  the  extracted  x-y 
location  within  ±10  cm,  and  the  depth  within  ±5  cm. 

4.5.4  Results 

Figure  18  and  Figure  19  show  the  distribution  of  depth  (defined  here  by  as  C  data  -  C  estimated)  where 
C=  X,  Y,  and  Depth)  and  lateral  positions  errors  respectively.  The  estimated  and  measured  depths 
discrepancies  for  TOI  targets  have  a  mean  of  1.6cm  and  a  standard  deviation  of  5  cm,  and  lateral 
position  errors  along  x  and  y  directions  have  means  0.25  cm  and  4.8  cm  standard  deviations  10  cm 
and  7.5  cm,  respectively.  Thus  overall  the  agreement  between  inverted  and  actual  values  of  depth  and 
lateral  positions  were  acceptable,  however,  Figure  18  and  Figure  19  show  that  estimated  depth  and 
locations  were  outside  the  required  criteria  specified  in  the  table  1.  We  believe  that  these  could  be 
related  to  surface  roughness  and  topology,  recording  accuracies  of  extrinsic  parameters  and  recovered 
targets  in  the  cell. 


Depth  Error  [cm] 
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Figure  18.  Histogram  of  depth  errors  (defined  as  (Z  estimated-Zdata ))  for  the  WMA  TOI  anomalies. 
The  distribution  shown  has  a  mean  of  1.6  cm  and  a  standard  deviation  of  5  cm.  There  is  good 
agreement  between  the  estimates  and  the  ground  truth. 
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Figure  19.  Histogram  of  lateral  (x,y)  errors  (defined  as  (left)  Xmeasured-  X estimated  and  (right)  ^measured-  ^estimated)  for  the 
WMA  TOI  targets.  Lateral  errors  in  x  and  y  directions  have  means  0.25  cm  and  4.8  cm  and  standard  deviations  10 
cm  and  7.5  cm,  respectively. 
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5.0  TEST  DESIGN 


The  only  required  test  at  the  WMA  site  entailed  collecting  target  characterization  training  data: 
Using  a  calibration  pit,  the  data-collection  team  made  a  series  of  static  measurements  of  example 
targets  at  several  depths  and  altitudes  in  order  to  cross-check  models,  confirm  Tx  and  Rx  polarity 
for  the  sensors,  and  characterize  the  so-called  Library  targets. 
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5.1  Demonstration  Schedule 


Table  2:  Demonstration  steps 


Preparation 

calibration 

Blind  data  set 

Post-survey 

analysis 

Tasks  and  demonstration  stages 

Feb  2015 

Mar-2015 

May-2015 

December-2015 

1.  Invert  all  calibration  data  sets 

X 

2.  Invert  MM  data  sets 

X 

3.  Build  custom  training  data  sets  and  request 
ground  truth  for  MM 

X 

4.  Redefine  MM  target  classifier  and  request 
additional  training  data  if  necessary 

X 

5.  Generate  final  dig  list  and  submit  to  IDA 

X 

6.  Conduct  retrospective  analysis  if  needed 

X 

REPORTING: 

7.  Draft  demo  plan 

X 

8.  Final  demo  plan 

X 

9.  Draft  demonstration  report 

X 

10.  Final  demonstration  report 

X 

Figure  20.  Gantt  chart  showing  schedule  of  detail  activities  to  be  conducted  during  the  WMA  ESTCP  MM 
data  inversion  and  classification  study. 
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6.0  DATA  ANALYSIS  PLAN 


We  analyzed  all  cued  data  for  the  MM  sensor  and  produced  prioritized  dig  lists  for  independent 
scoring. 

6.1  Extracting  Target  Locations 

Target  locations  were  determined  relative  to  the  sensor  coordinate  system  using  the  differential 
evolution  algorithm.  Object  responses  were  modeled  with  ONVMS.  The  combined  ONVMS-DE 
algorithm  was  run  for  single-  and  multi-target  cases  and  provided  target  locations. 

6.2  Extracting  Target  Intrinsic  Parameters 

6.2.1  Single  targets 

The  combined  ONVMS-DE  algorithm  yields  the  targets’  intrinsic  total  ONVMS,  which  we  used 
for  classification.  The  total  ONVMS  contains  three  moments,  m  (r) ,  M  (t) ,  and  m  ( t ) , 

xx  yy  zz 

along  the  primary  axes  in  the  target’s  own  reference  frame.  These  moments  are  similar  to  simple 
dipole  moment  components  but  carry  more  information,  accounting  for  the  targets’  inherent 
heterogeneities.  The  ONVMS-DE  algorithm  outputs  the  time-decay  curves  of  the  target’s  total 
ONVMS  tensor  M ..  (tk ) .  The  next  step  is  to  determine  the  time  decay  of  the  primary 

components  of  the  total  ONVMS  in  the  target’s  reference  frame.  While  this  can  be  done  by 
standard  diagonalization — i.e.,  finding  M(tk)  =  V (tk  )D(tk )VT  (tk ) ,  where  V(tk )  contains  the 
eigenvectors  of  M(tk ) — it  is  more  convenient  to  perform  a  joint  diagonalization, 

M(tk)  =  VD(tk)VT ,  where  now  the  eigenvectors  are  shared  by  all  time  channels;  this  allows  us  to 

extract  more  reliable  total  ONVMS  values  and  reduce  uncertainty.  The  resulting  temporal  decay 
of  the  total  principal  ONVMS  for  the  MM  anomalies  are  illustrated  in  Figure  6-Figure  10. 

6.2.2  Multi-target  cases 

A  similar  approach  is  carried  out  if  more  than  one  subsurface  target  is  expected.  The  DE  algorithm 
now  searches  for  the  locations  and  the  total  ONVMS  of  several  objects.  Such  multi-target  inversion 
is  crucial  in  the  field  for  cases  in  which  a  signal  from  a  UXO  is  mixed  with  EMI  signals  from 
nearby  clutter.  Our  two-target  inversion  code  yields  three  sets  of  location  and  total  ONVMS 
estimates:  one  for  Target  1,  one  for  Target  2,  and  a  combined  estimate  with  Targets  1  and  2 
represented  by  a  single  object.  (In  the  case  of  3-target  inversion,  seven  sets  of  data  are  expected: 
only  Target  1,  only  Target  2,  only  Target  3,  Targets  1  and  2  as  a  single  object,  Targets  2  and  3  as 
a  single  object,  Targets  1  and  3  as  a  single  object,  and  all  three  targets  acting  as  a  single  object.  In 
the  general  case  of  n  targets  one  expects  n  in  -  1)  +  1  sets  of  ONVMS  curves). 

6.3  Selection  of  Intrinsic  Parameters  for  Classification 

Most  UXO  are  bodies  of  revolution,  and  thus  the  two  secondary  polarizability  elements  are 
degenerate.  However,  live-site  UXO  discrimination  studies  have  repeatedly  shown  that  this 
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symmetry  can  be  compromised  due  to  low  SNR,  especially  for  small  or  deep  targets.  A  good 
classification  of  object  features  can  then  be  obtained  by  using  only  the  principal  component  of  the 
total  ONVMS  (  m __ ).  Furthermore,  to  limit  the  number  of  relevant  features  for  use  in  classification 

we  extract  parameters  exclusively  from  the  main  polarizability  both  to  represent  size 

M  O, )  and  wall  thickness  m  (t  )  /  M  (r, ) . 

zzx  ly  zzK  ny  zzK  ly 


6.4  Training 

Our  classification  approach  is  based  on  custom  training  data.  At  the  first  stage  of  the  process  we 
used  a  semi- supervised  clustering  technique  for  identifying  potential  site-specific  TOI.  Below  are 
the  basic  steps  performed  during  training  data  selection;  for  more  details  regarding  each  specific 
sensor  see  Section  2.3. 

(a)  The  targets’  intrinsic  features  (  Mrr  (t] ) ,  M__ (tn )  /  Mr_ (r, ) )  were  selected  from  the  extracted 

total  ONVMS;  n  was  chosen  based  on  feature  separation.  EMI  data  sets  of  all  anomalies, 
corresponding  to  single-and  multi-object  inversions,  were  produced. 

(b)  Initial  clustering  was  performed.  The  ground  truth  was  requested  for  all  targets  whose 
features  were  located  closest  to  the  corresponding  cluster  centroid  and  had  TOI-like 
ONVMS  features. 

(c)  Clusters  containing  at  least  one  TOI  were  identified,  and  a  smaller  domain  was  selected 
within  the  feature  space  for  further  interrogation. 

(d)  Additional  clustering  was  performed  within  the  selected  domain,  and  those  targets  with 
features  closest  to  the  corresponding  cluster  centroids  were  probed  for  ground  truth.  The 
clusters  with  at  least  one  identified  UXO  were  marked  as  suspicious.  The  total  ONVMS 
curves  were  inspected  within  the  selected  domain. 

(e)  All  targets  whose  features  (based  on  multi-object  inversion  and  library  matching)  fell 
inside  any  of  the  suspicious  clusters  were  used  to  train  the  statistical  classifier  and  the 
library-matching  procedure. 

6.5  Classification 

(f)  Probability  density  functions  were  created  for  single-  and  multi-target  scenarios. 

(g)  All  of  the  unknown  targets  were  scored  based  on  the  probability  density  functions. 

(h)  Dig  lists  were  produced  for  both  single-and  multi-object  cases  and  compared  to  each  other 
to  find  similarities  and  differences. 

(i)  All  items  were  further  analyzed  using  library  matching,  and  all  total  ONVMS  time-decay 
curves  were  inspected  visually. 

(j)  A  set  of  anomalies  were  identified  and  additional  training  data  sets  were  requested.  The 
new  information  was  incorporated  into  the  Gaussian  mixture  model  and  all  items  were  re¬ 
scored. 

(k)  Based  on  the  previous  steps,  a  classification  threshold  was  selected  and  a  final  dig  list 
produced. 
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6.6  Decision  Memo 


The  algorithms  used  to  select  training  data  and  to  perform  inversion  and  classifications  for  the 
WMA  test  are  described  in  Section  2.  Using  the  inversion,  clustering,  classification  and  data- 
requesting  procedures  outlined  above,  we  produced  a  ranked  anomaly  list  formatted  as  specified 
by  IDA  [20], 
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7.0COST  ASSESSMENT 


Time  and  resources  were  tracked  for  each  task  to  assess  the  cost  of  deploying  the  technology  at 
future  live  sites.  Note  that  some  of  the  costs  might  decrease  as  the  technology  matures  and  survey 
procedures  get  formalized.  A  time  and  cost  model  of  the  resources  spent  during  WMA  target 
classification  using  the  advanced  models  is  summarized  in  Table  3.  These  estimations  are  done  for 
a  geophysicist  with  salary  of  $90/hr. 


Table  3:  Cost  model  for  advanced  EMI  model  demonstration  at  the  WMA. 


Cost  Category 

Description 

Time 

Cost 

Preprocessing 

Time  required  to  perform  eigenvalue 
extraction,  check  data  quality,  and 
estimate  the  number  of  potential 
anomalies 

0.25  min/anomaly 

$0.375/Anomaly 

Parameter  extraction 

Time  required  to  run  code  and 
extract  target  feature  parameters 

0.25  min/anomaly 

$0.375/Anomaly 

Classifier  training 

Time  required  to  optimize  classifier 
design  and  train 

0.5  min/anomaly 

$0.75  /Anomaly 

Classification  and 

construction  of  a 
ranked  anomaly  list 

Time  required  to  classify  anomalies 
in  the  test  set  and  construct  the 
ranked  anomaly  list 

1.0  min/anomaly 

$  1.5/anomaly 

Reporting 

Time  required  to  generating  and 
documenting  classification  results 
and  writing  reports. 

2.0  min/anomaly 

$3. 0/anomaly 

Total 

4  min/anomaly 

$6/anomaly 
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8.0  MANAGEMENT  AND  STAFFING 


Fridon  Shubitidze  -  Principal  Investigator.  Responsible  for  MM  data  JD  analysis,  classification 
quality  check  and  reporting. 

Irma  Shamatava  -  White  River  Technologies.  Responsible  for  data  inversion  using  the  combined 
ONVMS/DE  algorithm  and  classification. 
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APPENDICES 


Appendix  A:  Health  and  Safety  Plan  (HASP) 

As  this  effort  does  not  involve  field  data  collection,  no  HASP  is  required. 
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Appendix  B:  Points  of  Contact 


Points  of  contact  (POCs)  involved  in  the  demonstration  and  their  contact  information  are  presented 
in  Table  4. 


Table  4:  Points  of  Contact  for  the  Advanced  EMI  Models  Demonstration. 


POINT  OF 
CONTACT 
Name 

ORGANIZATION 

Name 

Address 

Phone 

Fax 

E-mail 

Role  in  Project 

Dr.  Fridon 
Shubitidze 

White  River  Technologies 

115  Etna  Road 

Lebanon,  NH  03766 

Tel:  (603)  727-9549 
shubitidze  @  whiteri  vertech.  com 

PI 

Erik  Russell 

White  River  Technologies 

115  Etna  Road 

Lebanon,  NH  03766 

Tel:  (603)  678-8386 
russell  @  whiterivertech.com 

Project 

Coordination 

Dr.  Herb 
Nelson 

ESTCP  Program  Office 

901  North  Stuart  Street,  Suite  303 
Arlington,  VA  22203-1821 

Tel:  (571)  372-6400 
herb.nelson@nrl.navv.mil 

ESTCP  Munitions 
Management 
Program  Manager 
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